Moving state analysis device, moving state analysis method, and program

ABSTRACT

A moving state analysis device improves accuracy of moving state recognition by including a detection unit configured to detect, from image data associated with a frame, an object and a region of the object, for each of frames that constitute first video data captured in a course of movement of a first moving body, and a learning unit configured to learn a DNN model that takes video data and sensor data as input and that outputs a probability of each moving state, based on the first video data, a feature of first sensor data measured in relation to the first moving body and corresponding to a capture of the first video data, a detection result of the object and the region of the object, and information that indicates a moving state associated with the first video data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a U.S. 371 Application of International PatentApplication No. PCT/JP2019/018276, filed on 7 May 2019, whichapplication claims priority to and the benefit of JP Application No.2018-107057, filed on 4 Jun. 2018, the disclosures of which are herebyincorporated herein by reference in their entireties.

TECHNICAL FIELD

The present invention relates to a moving state analysis device, amoving state analysis method, and program.

BACKGROUND OF THE INVENTION

With the miniaturization of video imaging devices and power savings inGPS, gyroscopic sensors, and the like, it has become easy to recordactions of a user as a variety of data such as video, positioninginformation, and acceleration. Detailed analysis of user actions fromsuch data is useful for a variety of applications. For example, if asituation such as window shopping or crossing a pedestrian crossingcould be automatically recognized and analyzed, by using first-personview videos obtained through eyewear and the like and acceleration dataand the like obtained by wearable sensors, it would be useful in avariety of applications such as in personalizing services.

Conventionally, as technology for automatically recognizing a user'smoving state from sensor information, there exists technology forestimating a user's transportation mode from positioning information andvelocity information of GPS (Non-Patent Document 1). Additionally, usinginformation of acceleration and the like obtained from a smartphone,development of technology analyzing walking and jogging, climbing stairsand the like is also being undertaken (Non-Patent Document 2).

RELATED ART DOCUMENTS Non-Patent Documents

-   [Non-Patent Document 1] Zheng, Y., Liu, L., Wang, L., and Xie, X.:    Learning transportation mode from raw GPS data for geographic    applications on the web. In Proc. of World Wide Web 2008, pp.    247-256, 2008.-   [Non-Patent Document 2] Jennifer R. Kwapisz, Gary M. Weiss,    Samuel A. Moore: Activity Recognition using Cell Phone    Accelerometers, Proc. of SensorKDD 2010.

SUMMARY OF THE INVENTION Problem to be Solved by the Invention

However, as the conventional methods above use sensor information only,they have not been able to perform identification of a user's movingstate with consideration of video information. For example, when tryingto ascertain a user's moving state from wearable sensor data, even ifunderstanding that the user is walking, it is difficult from only sensordata to automatically recognize a specific situation of a user, such aswhether window shopping or crossing a pedestrian crossing.

On the other hand, in combining inputs of video data and sensor data,when using a simple classification model such as SVM (Support VectorMachine), which is one machine learning technology, highly accuratemoving state recognition was difficult due to differing levels ofabstraction with respect to the information from the video data and fromthe sensor data.

The present invention is made in view of the point above, and has theobjective of improving the accuracy of moving state recognition.

Means for Solving the Problem

To solve the above problem, a moving state analysis device includes adetection unit configured to detect, from image data associated with aframe, an object and a region of the object, for each of frames thatconstitute first video data captured in a course of movement of a firstmoving body, and a learning unit configured to learn a DNN model thattakes video data and sensor data as input and that outputs a probabilityof each moving state, based on the first video data, a feature of firstsensor data measured in relation to the first moving body andcorresponding to a capture of the first video data, a detection resultof the object and the region of the object, and information thatindicates a moving state associated with the first video data.

Advantage of the Invention

It is possible for the accuracy of moving state recognition to beimproved.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a hardware configuration of a movingstate recognition device 10 in an embodiment of the present invention.

FIG. 2 is a diagram illustrating a functional configuration of themoving state recognition device 10 in a learning phase.

FIG. 3 is a diagram of a functional configuration of the moving staterecognition device 10 in a recognition phase.

FIG. 4 is a flowchart for explaining an example of a processingprocedure executed by the moving state recognition device 10 in thelearning phase.

FIG. 5 is a diagram illustrating a configuration example of a video dataDB 121.

FIG. 6 is a diagram illustrating a configuration example of a sensordata DB 122.

FIG. 7 is a diagram illustrating a configuration example of anannotation DB 124.

FIG. 8 is a flowchart for explaining an example of a processingprocedure executed by the moving state recognition device 10 in therecognition phase.

FIG. 9 is a flowchart for explaining an example of a processingprocedure executed by a video data preprocessing unit 11.

FIG. 10 is a diagram illustrating an example of image data in each framegenerated from video data.

FIG. 11 is a flowchart for explaining an example of a processingprocedure executed by a sensor data preprocessing unit 12.

FIG. 12 is a flowchart for explaining an example of a processingprocedure executed by an object detection unit 13.

FIG. 13 is a diagram illustrating an example of an object detectionresult.

FIG. 14 is a flowchart for explaining an example of a processingprocedure executed by a spatial feature embedding unit 14.

FIG. 15 is a diagram illustrating an example of spatial feature embeddeddata.

FIG. 16 is a flowchart for explaining an example of a processingprocedure of an embedding process of spatial features.

FIG. 17 is a diagram illustrating an example of a structure of a DNNmodel constructed by a DNN model constructing unit 15.

FIG. 18 is a flowchart for explaining an example of a processingprocedure executed by a moving state recognition model learning unit 16.

FIG. 19 is a diagram illustrating an example of model parameters.

FIG. 20 is a flowchart for explaining an example of a processingprocedure executed by a moving state recognition unit 17.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, an embodiment of the present invention is explained basedon the drawings. FIG. 1 is a diagram illustrating a hardwareconfiguration of a moving state recognition device 10 in an embodimentof the present invention. The moving state recognition device 10 of FIG.1 includes a drive device 100, an auxiliary storage device 102, a memorydevice 103, a CPU 104, and an interface device 105 and the like mutuallyinterconnected through a bus B.

The programs implementing the processing by the moving state recognitiondevice 10 are provided by a recording medium 101 of a CD-ROM or thelike. When the recording medium 101 in which the programs are stored isset in the drive device 100, the programs are installed in the auxiliarystorage device 102 from the recording medium 101 through the drivedevice 100. However, the programs are not necessarily installed from therecording medium 101, and may be downloaded from another computerthrough a network. The auxiliary storage device 102 stores necessaryfiles, folders, and such along with storing the installed programs.

The memory device 103 reads out and stores programs from the auxiliarystorage device 102, in response to a program start command. The CPU 104executes functions related to the moving state recognition device 10 inaccordance with programs stored in the memory device 103. The interface105 is used as an interface for connecting to a network.

FIG. 2 is a diagram illustrating a functional configuration example ofthe moving state recognition device 10 in a learning phase. With respectto the learning phase, the moving state recognition device 10 includes avideo data preprocessing unit 11, a sensor data preprocessing unit 12,an object detection unit 13, a spatial feature embedding unit 14, a DNNmodel constructing unit 15, and a moving state recognition modellearning unit 16. These units are actualized by processing executed inthe CPU 104 of the one or more programs installed in the moving staterecognition device 10. The moving state recognition device 10,additionally, with respect to the learning phase, uses a video data DB121, a sensor data DB 122, an object detection model DB 123, anannotation DB 124, a DNN model DB 125, and the like. Each of these DB(Data Base) can be actualized using the auxiliary storage device 102, astorage device connectable to the moving state recognition device 10through a network, or the like.

In the learning phase, the moving state recognition device 10 outputs amoving state recognition DNN (Deep Neural Network) model usinginformation stored in the respective DBs.

In the video data DB 121, 1 or more of video data is stored. In thepresent embodiment, 1 video data is regarded as moving image datacaptured over the course of movement of a vehicle, in accordance with acamera (for example, a drive recorder and the like) mounted in thevehicle.

In the sensor data DB 122, sensor data measured by each type of sensor,in the vehicle in which the camera captures the video data, is stored.

Note that in the video data DB 121 and the sensor data DB 122, the videodata or the sensor data is stored correlated with a common data ID suchthat correspondence of corresponding video data and sensor data can beunderstood. Corresponding video data and sensor data refers to videodata and sensor data that are captured and measured in a same vehicleand same time interval. For example, when a video data-sensor data pairis input by a system operator, the video data may be stored in the videodata DB 121 and the sensor data may be stored in the sensor data DB 122,with an ID as a data ID uniquely identifying the pair being assigned tothe input video data and sensor data.

In the object detection model DB 123, a model structure of the trained(learned) object detection model and parameters are stored. Here, objectdetection refers to detecting the general name of an object (objectname) projected in one image along with a boundary region projected bythe object. For the object detection model, it is possible to use apublicly known model such as an SVM trained by image features such asfrom HOG, or a DNN such as YOLO. Note that there is detail regarding HOGin “Dalal, Navneet and Triggs, Bill: Histograms of Oriented Gradientsfor Human Detection. In Proc. of Computer Vision and Pattern Recognition2005, pp. 886-893, 2005”. Additionally, there is detail regarding YOLOin “J. Redmon, S. Divvala, R. Girshick and A. Farhadi: You Only LookOnce: Unified, Real-Time Object Detection, Proc. of Computer Vision andPattern Recognition 2016, pp. 779-788, 2016”.

In the annotation DB 124, annotation names corresponding to respectivedata IDs are stored. Here, an annotation name is assumed to be acharacter string explaining a situation corresponding to video data, andis applicable to “car near-miss”, “bicycle near-miss”, or the like.Similarly to the video data DB 121 or the sensor data DB 122, for theannotation DB 124 also, annotation names corresponding to respectivedata IDs may be input by a system operator, and the input results may bestored in the annotation DB 124.

FIG. 3 is a diagram illustrating a functional configuration example ofthe moving state recognition device 10 in a recognition phase. In FIG.3, parts that are the same as in FIG. 2 are denoted with the samesymbols, and the explanation thereof is omitted.

In the recognition phase, the moving state recognition device 10includes a video data preprocessing unit 11, a sensor data preprocessingunit 12, an object detection unit 13, a spatial feature embedding unit14, and a moving state recognition unit 17. The moving state recognitionunit 17 is actualized by processing executed in the CPU 104 of the oneor more programs installed in the moving state recognition device 10.The moving state recognition device 10 additionally uses an objectdetection model DB 123 and a DNN model DB 125 in the recognition phase.

In the recognition phase, the moving state recognition device 10 outputsa recognition result corresponding to input video data and input sensordata, using information stored in each DB. The input video data is, forexample, video data that differs from the video data stored in the videodata DB 121 and that is captured in the same vehicle or a differentvehicle as the vehicle that captured the video data stored in the videodata DB 121. Additionally, the input sensor data refers to sensor datathat is measured in the same vehicle as the vehicle that captured theinput video data and that is measured in the same time interval as thetime interval of capturing the corresponding video data.

Note that different computers may be used for the learning phase and therecognition phase.

Below, the processing procedure executed by the moving state recognitiondevice 10 is described. FIG. 4 is a flowchart for explaining an exampleof a processing procedure executed by the moving state recognitiondevice 10 in the learning phase.

In Step S100, the video data preprocessing unit 11 acquires each videodata from the video data DB 121, and executes preprocessing for eachvideo data.

FIG. 5 is a diagram illustrating a configuration example of a video dataDB 121. As illustrated in FIG. 5, in the video data DB 121, video datacorresponding to data IDs so as to be linked with sensor data is stored.Video data is stored, for example, in files compressed by an Mpeg4format and the like. Note that video data of differing data IDs is, forexample, video data of which the vehicles that captured the data differ,or video data of which the capturing intervals differ.

As a result of the preprocessing, each video data is converted to aformat that is easily usable by the DNN model. Hereinafter, the datagenerated by preprocessing is referred to as “preprocessed video data”.Note that detail of the preprocessing of the video data is describedlater.

Next, the sensor data preprocessing unit 12 acquires each sensor datafrom the sensor data DB 122, and executes preprocessing for each sensordata. (S110).

FIG. 6 is a diagram illustrating a configuration example of a sensordata DB 122. Each sensor data is assigned an individual sequence ID, andis stored corresponding to a data ID so as to be linked with the videodata. Each sensor data includes elements of date-and-time, latitude andlongitude, X-axis acceleration and Y-axis acceleration, and the like.

As a result of the preprocessing, each sensor data is converted to aformat that is easily usable by the DNN model (later-described “featurevector”). Hereinafter, the data generated by preprocessing is referredto as “preprocessed sensor data”. Note that detail of the preprocessingof the sensor data is described later.

Next, the object detection unit 13 receives preprocessed video data fromthe video data preprocessing unit 11, and, by using the object detectingmodel stored in the object detection model DB 123, from the preprocessedvideo data, detects the object name and region of each object includedin the video data (S120). Detail of this process is described later.

Next, the spatial feature embedding unit 14 receives an object detectionresult from the object detection unit 13, and generates spatial featureembedded data of the object detection result (S130). Detail of theprocess and detail of the spatial feature embedded data are describedlater.

Next, the DNN model constructing unit 15 constructs the DNN model(S140). Detail of the DNN model is described later.

Next, the moving state recognition model learning unit 16 receivespreprocessed video data from the video data preprocessing unit 11,preprocessed sensor data from the sensor data preprocessing unit 12,spatial feature embedded data from the spatial feature embedding unit14, a DNN model from the DNN model constructing unit 15, and annotationdata from the annotation DB 124, learns the DNN model, and outputs theDNN model as the learning result to the DNN model DB 125 (S150).

FIG. 7 is a diagram illustrating a configuration example of theannotation DB 124. As illustrated in FIG. 7, in the annotation DB 124,annotation names associated with respective data IDs are recorded.

FIG. 8 is a flowchart for explaining an example of a processingprocedure executed by the moving state recognition device 10 in therecognition phase.

In step S200, the video data preprocessing unit 11 receives video dataas input, and for the video data, executes preprocessing similarly tostep S100. Hereinafter, the data generated by preprocessing is referredto as “preprocessed video data”.

Next, the sensor data preprocessing unit 12 receives sensor data asinput, and for the sensor data, executes preprocessing similarly to stepS110 (S210). Hereinafter, the data generated by preprocessing isreferred to as “preprocessed sensor data”.

Next, the object detection unit 13 receives preprocessed video data fromthe video data preprocessing unit 11, and, by using the object detectingmodel stored in the object detection model DB 123, from the preprocessedvideo data, detects the object name and region of each object includedin the video data (S220).

Next, the spatial feature embedding unit 14 receives an object detectionresult from the object detection unit 13, and generates spatial featureembedded data of the object detection result (S230). Although detail ofthe spatial feature embedded data is described later, one example of thespatial feature embedded data is data that indicates features ofrespective regions (region of a video [image]) in which the objectsappear.

Next, a moving state recognition unit 17 receives preprocessed videodata from the video data preprocessing unit 11, preprocessed sensor datafrom the sensor data preprocessing unit 12, spatial feature embeddeddata from the spatial feature embedding unit 14, and a learned DNN modelfrom the DNN model DB 125, calculates a moving state recognition resulton a basis of these, and outputs the calculated result (S240).

Next, detail of step S100 of FIG. 4 and step S200 of FIG. 8 isexplained. FIG. 9 is a flowchart for explaining an example of aprocessing procedure executed by the video data preprocessing unit 11.The processing procedure of FIG. 9 is executed for each data ID in thelearning phase case and executed for one input video data in therecognition phase case.

In step S300, the video data preprocessing unit 11 acquires video dataof a processing target (Hereinafter, this is referred to as “targetvideo data”.). The video data of the processing target in the learningphase case is video data associated with the data ID of the processingtarget, from among the set of video data stored in the video data DB121. The video data of the processing target in the recognition phase isone input video data.

Next, the video data preprocessing unit 11 converts the image data ineach frame of the target video data to image data represented by pixelvalues of a predetermined vertical size×predetermined horizontal size×3channels (RGB) (S310). For example, the predetermined vertical size maybe assumed to be 100 pixels and the predetermined horizontal size may beassumed to be 200 pixels.

FIG. 10 is a diagram illustrating an example of image data in each framegenerated from video data. As illustrated in FIG. 10, each image data isassociated with the corresponding data ID in the source video data, theframe number associated with the image data of the video data, and thetimestamp information of the frame of the video data.

Note that data ID is unnecessary in the recognition phase. In therecognition phase, as video data is input with sensor data correspondingto the video data, the correspondence relationship of the video data andthe sensor data is clear.

Next, the video data preprocessing unit 11 performs sampling by aconstant frame interval (interval of N frames) of the target video dataframes, to reduce redundant data (S320). As a result, a part of theframes is removed.

Next, in order for the image data of each of the sampled frames to beeasily usable in the DNN model, the video data preprocessing unit 11normalizes each pixel value of the image data of each of gear the frames(S330). For example, normalization is performed by dividing each pixelvalue by the greatest value a pixel can take, such that the range ofeach of the pixel values is scaled down to a range of 0 to 1.

Next, the video data preprocessing unit 11 outputs video datarepresented as an image data sequence consisting of image data of eachframe normalized in step S330 and date-and-time information of eachframe of the video data after sampling (S340). The video data ispreprocessed video data of the target video data. Additionally, thedate-and-time information of each frame can be obtained by adding thetimestamp (FIG. 10) of each frame to the date and time of startingcapture (not illustrated in FIG. 5), which is stored for each video datain the video data DB 121. Note that, in the learning phase case, theoutput destination is the object detection unit 13 and the moving staterecognition model learning unit 16. In the recognition phase case, theoutput destination is the object detection unit 13 and the moving staterecognition unit 17.

Next, detail of step S110 of FIG. 4 and step S210 of FIG. 8 isdescribed. FIG. 11 is a flowchart for explaining an example of aprocessing procedure executed by the sensor data preprocessing unit 12.The processing procedure of FIG. 11 is executed for each data ID in thelearning phase case and executed for the input sensor data in therecognition phase case.

In step S400, the sensor data preprocessing unit 12 acquires sensor dataof a processing target (Hereinafter, this is referred to as “targetsensor data”.). The sensor data of the processing target in the learningphase case is sensor data associated with the data ID of the processingtarget, from among the set of sensor data stored in the sensor data DB122. The sensor data of the processing target in the recognition phaseis the input sensor data.

Next, the sensor data preprocessing unit 12 normalizes each measuredvalue of each sensor data of the processing target, so that the sensordata is easily usable in the DNN model (S410). For example, for each oflatitude, longitude, and X-axis acceleration and Y-axis acceleration,the sensor data preprocessing unit 12 normalizes such that the meanvalue is 0 and the standard deviation is 1 for all the sensor data ofthe processing target.

Next, the sensor data preprocessing unit 12 generates a result ofcombining each of the normalized values of the respective sensor data,as a feature vector of the respective sensor data (S420). For example,if the measured values included in the sensor data are the four oflatitude, longitude, and X-axis acceleration and Y-axis acceleration,the feature vector is constituted by the normalized latitude, thenormalized longitude, and the normalized X-axis acceleration andnormalized Y-axis acceleration. Note that each feature vector isassociated with the corresponding sensor data.

Next, the sensor data preprocessing unit 12 outputs the feature vectorof the respective data, and the date-and-time information of eachfeature vector. The date-and-time information of each feature vector isthe value of “date-and-time” stored in the sensor data DB 122 of thesensor data that is the source for the feature vector. Note that in thelearning phase case, the output destination is the moving staterecognition model learning unit 16. In the recognition phase case, theoutput destination is the moving state recognition unit 17.

Next, detail of step S120 of FIG. 4 and step S220 of FIG. 8 isdescribed. FIG. 12 is a flowchart for explaining an example of aprocessing procedure executed by the object detection unit 13.

In step S500, the object detection unit 13 receives preprocessed videodata from the video data preprocessing unit 11.

Next, the object detection unit 13 acquires a learned object detectionmodel (model structure and parameters) from the object detection modelDB 123 (S510).

Next, the object detection unit 13, using the object detection model,for each frame of the preprocessed video data, executes an objectdetection process with respect to the image data associated with theframe (S520).

FIG. 13 is a diagram illustrating an example of an object detectionresult. As illustrated in FIG. 13, the object detection result, for eachobject detected from the image data associated with the correspondingframe, includes the data ID of the video data in which the framebelongs, the frame number of the frame, the date-and-time information ofthe frame, a designation representing the object, and information ofcoordinates (left edge, top edge, right edge, bottom edge) and the likerepresenting the boundary region in which the object in the image datais detected.

Next, the object detection unit 13 outputs the object detection result(FIG. 13) to the spatial feature embedding unit 14 (S530).

Next, detail of step S130 of FIG. 4 and step S230 of FIG. 8 isexplained. FIG. 14 is a flowchart for explaining an example of aprocessing procedure executed by the spatial feature embedding unit 14.

In step S600, the spatial feature embedding unit 14 receives the objectdetection result from the object detection unit 13.

Next, the spatial feature embedding unit 14 executes an embeddingprocess of spatial features to grid space with respect to objectsincluded in the object detection result, in accordance with a previouslydefined grid size (S610). As a result of the spatial feature embeddingprocess, spatial feature embedded data is generated.

FIG. 15 is a diagram illustrating an example of spatial feature embeddeddata. As illustrated in FIG. 15, the spatial feature embedded data isgenerated for each frame. In other words, step S610 is executed for eachgroup of objects of common frame number in FIG. 13. With respect to onespatial feature embedded data, a configuration of verticalgrid×horizontal grid×detectable number of objects is included. Note thatdetail of an embedding process of spatial features is described later.

Next, the spatial feature embedding unit 14 outputs the respectivespatial feature embedded data of each frame and the date-and-timeinformation corresponding to the respective spatial feature embeddeddata (S620). The date-and-time information corresponding to the spatialfeature embedded data can be specified based on the date and time ofstarting capture of the video data and the timestamp of FIG. 15. Notethat in the learning phase case, the output destination is the movingstate recognition model learning unit 16. In the recognition phase case,the output destination is the moving state recognition unit 17.

Next, detail of step S610 is described. FIG. 16 is a flowchart forexplaining an example of a processing procedure of an embedding processof spatial features. The processing procedure of FIG. 16 is executed foreach frame (for each record of common frame number) of the objectdetection result (FIG. 13).

In step S700, the spatial feature embedding unit 14 initializes the dataconfiguration of the embedding destination, and related variables.

Here, with respect to the image size of the image data of each frame ofthe preprocessed video data, H represents vertical and W representshorizontal. Note that, as the image size of each image data isstandardized by preprocessing, H and W are common to each image data.Additionally, a previously defined vertical grid size is represented asGh, horizontal grid size as Gw, and the number of detectable objecttypes in the learned object detection model as O. The matrixG∈R^(Gh×Gw×0)←0 of the embedding destination is initialized.Additionally, the vertical grid unit width Sh for the grid structure ofthe embedding destination of the spatial features is obtained bySh←H/Gh, and the horizontal grid unit width Sw by Sw←W/Gw.

Next, the spatial feature embedding unit 14 receives the objectdetection result of the frame of the processing target (S710). Here, thegroup of objects detected from one frame of the processing target isrepresented by {o₁, o₂, . . . , o_(N)}. N is the number of objectsdetected from the frame, and changes by frame. The ID identifying thename of each detectable object is represented by o_(n).id∈[1, 2, . . . ,O], and the coordinates of the left edge, top edge, right edge, andbottom edge representing the boundary region of each object detectedfrom the frame are respectively represented by o_(n).left, o_(n).top,o_(n). right, o_(n).bottom.

Next, by the spatial feature embedding unit 14, for each object n∈[1, 2,. . . , N] detected from the image data of the processing target frame,the left edge index left, the top edge index top, the right edge indexright, and the bottom edge index bottom, with respect to the definedgrid size,

$\begin{matrix}{ {left}arrow\lceil \frac{o_{n} \cdot {left}}{Sw} \rceil , {top}arrow\lceil \frac{o_{n} \cdot {top}}{Sh} \rceil , {right}arrow\lceil \frac{o_{n} \cdot {right}}{Sw} \rceil , {bottom}arrow\lceil \frac{o_{n} \cdot {bottom}}{Sh} \rceil } & \lbrack {{Math}.1} \rbrack\end{matrix}$are respectively calculated (S720).

Next, the spatial feature embedding unit 14 calculates the score of thegrid cell g_(i,j,k) as follows, with respect to the indexi(top≤i≤bottom), j(left≤j≤right), k←o_(n).id (S730). Note that the gridcell g_(i,j,k) is the smallest unit of the matrix G of the embeddingdestination. Here, three types of score calculation formulas areillustrated.

Calculation Example 1) Count the number of appearances of the object k,for each grid cell g_(i,j,k).g _(i,j,k) ←g _(i,j,k)+1

Calculation Example 2) Consider appearance locations of object k byGaussian kernel for each grid cell g_(i,j,k).g _(i,j,k) ←g _(i,j,k)+exp(−|i−m| ²)+exp(−|j−c|²),m←(top+bottom)/2,c←(left+right)/2

Calculation Example 3) Consider area ratio of boundary region of objectk to image size for each grid cellg _(i,j,k) ←g _(i,j,k)+(height×width)/(H×W),height←o _(k).bottom−o_(k).top,width←o _(k).right−o _(k).leftThe obtained G by g_(i,j,k) being calculated in this manner constitutesthe spatial feature embedded data of the frame.

Next, the DNN model constructed (generated) in step S140 of FIG. 4 isdescribed. FIG. 17 is a diagram illustrating an example of a structureof a DNN model constructed by the DNN model constructing unit 15.

The DNN model obtains the probability of each moving state as output,receiving, as input, the preprocessed video data, the preprocessedsensor data (that is, the feature vector of the sensor data)corresponding to the video data, and the spatial feature embedded datagenerated based on the video data. As illustrated in FIG. 17, the DNNnetwork is constituted from units such as a convolutional layer A, afully-connected layer A, a convolutional layer B, a fully-connectedlayer B, a fully-connected layer C, an LSTM (Long-short-term-memory),and a fully-connected layer D.

Convolutional layer A extracts features from the image matrix (that is,the images set of three channels) of 1 frame. Convolutional layer A, forexample, convolutes the image by a 3×3 filter, extracts (max pooling)the greatest value in the specified rectangle, and the like. For theconvolutional layer, it is possible to use a network structure or apreviously learned parameter that is publicly known such as from AlexNet(└Krizhevsky, A., Sutskever, I. and Hinton, G. E.: ImageNetClassification with Deep Convolutional Neural Networks, pp. 1106-1114,2012.┘)) The fully-connected layer A further abstracts the featuresobtained from the convolutional layer A. The fully-connected layer A,for example, using a sigmoid activation function, ReLu activationfunction, or the like, applies a non-linear transformation to thefeatures of the input.

The convolutional layer B extracts features from the spatial featureembedded data of the object detection result. The convolutional layer Balso performs processing similar to the convolutional layer A. In otherwords, the convolutional layer B, for example, convolutes the spatialfeature embedded data of the object detection result by a 3×3 filter,extracts (max pooling) the greatest value in the specified rectangle,and the like.

The fully-connected layer B further abstracts the features obtained fromthe convolutional layer B. The fully-connected layer B, similarly to thefully-connected layer A, applies a non-linear transformation to theinput features.

The fully-connected layer C abstracts the feature vector of the sensordata to a level equivalent to the image features obtained from thefully-connected layer A. The fully-connected layer C, similarly to thefully-connected layer A, applies a non-linear transformation to thefeature vector of the input. Note that abstraction to a level equivalentto the image features refers to aligning the scale (range from upperlimit to lower limit) of the value of each dimension of the sensor datafeature vector, and the number of the dimensions of the feature vector,with the scale of the value of each dimension of the vector indicatingthe image features, and the number of the dimensions of the vectorindicating the image features.

An LSTM further abstracts, as sequential features, the 3 abstractedfeatures output from the fully-connected layer A, the fully-connectedlayer B, and the fully-connected layer C. Specifically, the LSTMsequentially receives sequential data of features output from thefully-connected layer A, the fully-connected layer B, and thefully-connected layer C, and while circulating past abstractedinformation, repeatedly applies a non-linear transformation to thesequential data. A publicly known network structure ([Felix A. Gers,Nicol N. Schraudolph, and Juergen Schmidhuber: Learning precise timingwith LSTM recurrent networks. Journal of Machine Learning Research, vol.3, pp. 115-143, 2002.]) in which an LSTM is equipped with a forget gatecan also be used.

The fully-connected layer D calculates a probability vector with respectto each moving state, by applying the sequential features abstracted byLSTM to a vector of which the dimension is the number of types of movingstates of interest. The fully-connected layer D calculates theprobability vector with respect to each moving state, by applying anon-linear transformation such that the sum of all elements of the inputfeatures is equal to 1, using a softmax function and the like.

The output layer outputs a probability vector with respect to eachmoving state. For example, information such as “non near-miss: 10%, carnear-miss: 5%, bicycle near-miss: 70%, motorcycle near-miss: 5%,pedestrian near-miss: 5%, other than above: 5%” is output.

Note that as the structure of the DNN model is not affected by the videodata and the sensor data, the DNN model may be constructed beforehandand stored in a database.

Next, detail of step S150 of FIG. 4 is described. FIG. 18 is a flowchartfor explaining an example of a processing procedure executed by themoving state recognition model learning unit 16.

In step S800, the moving state recognition model learning unit 16associates the image data of the frame of the preprocessed video data,the feature vector of the sensor data, and the spatial feature embeddeddata in 1:1:1, based on the date-and-time information of each frame ofthe preprocessed video data, the date-and-time information of the sensordata, and the date-and-time information of the spatial feature embeddeddata. For example, data of matching date-and-time information or data ofwhich the timing indicated by the data-and-time information is theclosest is associated together.

Next, the moving state recognition model learning unit 16 receives thenetwork structure (DNN model) as illustrated in FIG. 17 from the DNNmodel constructing unit 15 (S810).

Next, the moving state recognition model learning unit 16 initializesthe model parameters of each unit in the network (DNN model) (S820). Forexample, each model parameter is initialized by a random number from 0to 1.

Next, the moving state recognition model learning unit 16 updates modelparameters using the preprocessed video data, the feature vectorobtained by the preprocessing for the sensor data corresponding to thevideo data, the spatial feature embedded data based on the video data,and the annotation names corresponding to the video data (S830). Theannotation names corresponding to the video data refer to annotationnames stored in the annotation DB that are associated with the same dataID of the video data.

Specifically, the moving state recognition model learning unit 16, foreach set of image data, feature vector, and spatial feature embeddeddata associated in step S800, calculates the model parameters of the DNNmodel, using the image data, the feature vector, the spatial featureembedded data, and the annotation names stored in the annotation DB 124associated with the same data ID of the video data to which the imagedata belongs. In more detail, the moving state recognition modellearning unit 16, for each set, determines a probability vector of theoutput obtained with respect to the image data, the feature data, andthe spatial feature embedded data, and a binary vector obtained from theannotation names, and optimizes the model parameters using a publiclyknown technique such as back propagation so as to minimize thecross-entropy error of the probability vector and the binary vector.

Next, the moving state recognition model learning unit 16 outputs themoving state recognition DNN model (network structure and modelparameters), and stores the output result in the DNN model DB 125(S840).

FIG. 19 is a diagram illustrating an example of model parameters. Asillustrated in FIG. 19, parameters are stored as matrices or vectors ineach layer. Additionally, for the 1, 2, 3 of the output layer,annotation names (text indicating the moving state) corresponding torespective element numbers of the probability vector for thecorresponding moving states calculated in the fully-connected layer Dare stored.

Next, detail of step S240 of FIG. 8 is described. FIG. 20 is a flowchartfor explaining an example of a processing procedure executed by themoving state recognition unit 17.

In step S900, the moving state recognition unit 17 receives thepreprocessed video data from the video data preprocessing unit 11,receives the preprocessed sensor data from the sensor data preprocessingunit 12, and receives the spatial feature embedded data of the objectdetection result from the spatial feature embedding unit 14.

Next, the moving state recognition unit 17 acquires a learned movingstate recognition DNN model from the DNN model DB 125 (S910).

Next, the moving state recognition unit 17 calculates the probabilitywith respect to each moving state from the video data, the sensor data,and the spatial feature embedded data received in step S900 (S920).

Next, the moving state recognition unit 17 outputs the annotation nameassociated with the moving state with the highest probability (S930).

Note that, in the present embodiment, although an example of video dataand measured data being captured or measured in a vehicle has beendescribed, the present embodiment may be applied to video data andmeasured data captured in relation to the movement of a person. In thiscase, the video data may be first-person view video data obtainedthrough eyewear and the like worn by a person, and the sensor data maybe sensor data obtained by wearable sensors embodied by the person.Additionally, video data captured of a person from a third-person view,and sensor data obtained from sensors around the person, may be appliedto the present embodiment. Additionally, the present embodiment may beapplied in relation to a moving body other than a vehicle and a person.

As described above, according to the present embodiment, constructingand learning a model using video data in addition to sensor data andusing the acquired model in moving state recognition enablesidentification of moving states that could not be identifiedconventionally. As a result, the accuracy of moving state recognitioncan be improved.

Additionally, a user's moving state can be identified with high accuracyby use of a moving state recognition DNN model provided with aconvolutional layer that can handle effective image features for movingstate recognition, a fully-connected layer that can abstract featureswith respect to a suitable abstraction level, and an LSTM that canefficiently abstract sequence data.

Additionally, by using as input data an effective object detectionresult for moving state recognition, it becomes possible to identify auser's moving state with high accuracy.

Additionally, by feature embedding a detection boundary region of anobject detection result into grid space with use as input data,consideration for detection location of the object is made possible, andidentification of a user's moving state with high accuracy is madepossible. Additionally, it becomes possible to consider the number ofdetections of the object, the area ratio of the detected region, thecenter of gravity of the detected region and the like in accordance withthe calculation method at the time of spatial feature embedding, and itbecomes possible to identify a user's moving state with high accuracy.

Further, by using all of the objects detected in the object detectionresult as one structured data of embedded spatial features, it becomespossible to reduce the computational cost as compared to a method oftreating all of the objects as respective input data.

Note that, in the present embodiment, the moving state recognitiondevice 10 is an example of a moving state analysis device. The videodata stored in the video data DB 121 is an example of first video data.The sensor data stored in the sensor data DB 122 is an example of firstsensor data. The video data input in the recognition phase is an exampleof second video data. The sensor data input in the recognition phase isan example of second sensor data. The object detection unit 13 is anexample of a detection unit. The moving state recognition model learningunit 16 is an example of a learning unit. The moving state recognitionunit 17 is an example of a calculation unit. The spatial featureembedding unit 14 is an example of a generating unit. An annotation nameis an example of information that indicates a moving state.

An embodiment of the present invention has been described as above;however, the present invention is not limited to the above-describedspecific embodiment, and various variations and modifications may bemade within the scope of the gist of the present invention recited inthe scope of the patent claims.

The present application is based on and claims priority to Japanesepatent application No. 2018-107057 filed on Jun. 4, 2018, the entirecontents of which are hereby incorporated by reference.

DESCRIPTION OF SYMBOLS

-   10 moving state recognition device-   11 video data preprocessing unit-   12 sensor data preprocessing unit-   13 object detection unit-   14 spatial feature embedding unit-   15 DNN model constructing unit-   16 moving state recognition model learning unit-   17 moving state recognition unit-   100 drive device-   101 recording medium-   102 auxiliary storage device-   103 memory device-   104 CPU-   105 interface device-   121 video data DB-   122 sensor data DB-   123 object detection model DB-   124 annotation DB-   125 DNN model DB-   B bus

What is claimed is:
 1. A moving state analysis device comprising: amemory; and a processor coupled to the memory and configured to: detect,from image data associated with a frame, an object and a region of theobject, for each of frames that constitute first video data captured ina course of movement of a first moving body; and learn a deep neuralnetwork model that takes video data and sensor data as input and thatoutputs a probability of each moving state associated with a movingbody, based on a combination of at least: the first video data, afeature of first sensor data including positioning information measuredin relation to the first moving body at a time of capturing the firstvideo data, a detection result of the object and the region of theobject appearing in the first video data, and information indicating amoving state associated with the first moving body relative to theobject detected in the first video data.
 2. The moving state analysisdevice according to claim 1, wherein the processor is configured to:detect, from image data associated with a frame, an object and a regionof the object, for each of frames that constitute second video datacaptured in a course of movement of a second moving body; and calculatea probability of each moving state, with respect to the second videodata, by inputting into the deep neural network model a combination ofat least: the second video data, second sensor data measured in relationto the second moving body at a time of capturing the second video data,and a detection result of the object and the region of the objectdetected from the image data associated with the frame from which thesecond video data is constituted, the deep neural network model beingread and executed by the hardware processor.
 3. The moving stateanalysis device according to claim 2, wherein the processor isconfigured to: generate, based on the detection result of the object andthe region of the object, data indicating for each object a feature of aregion in which the object appears; learn the deep neural network modelbased on the generated data in relation to the first video data; andcalculate a probability of each moving state associated with the movingbody, based on the generated data in relation to the second video data.4. The moving state analysis device according to claim 1, wherein thepositioning information includes either latitude or longitude of alocation corresponding to the first moving body.
 5. The moving stateanalysis device according to claim 1, the processor further configuredto capture the first sensor data using a sensor.
 6. The moving stateanalysis device according to claim 1, wherein the first video data andthe first sensor data are captured on the first moving body, and whereinthe first moving body is distinct from the object.
 7. Acomputer-implemented method for analyzing a moving state, the methodcomprising: detecting an object and a region of the object from imagedata associated with a frame, for each of frames that constitute firstvideo data captured in a course of movement of a first moving body; andlearning a deep neural network model that takes video data and sensordata as input and that outputs a probability of each moving stateassociated with a moving body, based on a combination of at least: thefirst video data, a feature of first sensor data including positioninginformation measured in relation to the first moving body at a time ofcapturing the first video data, a detection result of the object and theregion of the object appearing in the first video data, and informationindicating a moving state associated with the first moving body relativeto the object detected in the first video data.
 8. The moving stateanalysis method executed by a computer according to claim 7, furthercomprising: detecting an object and a region of the object from imagedata associated with a frame, for each of frames that constitute secondvideo data captured in a course of movement of a second moving body; andcalculating a probability of each moving state, with respect to thesecond video data, by inputting into the deep neural network model acombination of at least: the second video data, second sensor datameasured in relation to the second moving body at a time of capturingthe second video data, and a detection result of the object and theregion of the object detected from the image data associated with theframe from which the second video data is constituted.
 9. The movingstate analysis method executed by a computer according to claim 8,further comprising: generating data indicating for each object a featureof a region in which the object appears, based on the detection resultof the object and the region of the object, wherein in the learning, thedeep neural network model is learned based on data generated in thegenerating in relation to the first video data; and in the calculating,a probability of each moving state associated with the moving body iscalculated based on data generated in the generating in relation to thesecond video data.
 10. A non-transitory computer-readable recordingmedium having a program that causes a computer to execute the movingstate analysis method of claim
 7. 11. The non-transitorycomputer-readable recording medium according to claim 10, wherein thepositioning information includes either latitude or longitude of alocation corresponding to the first moving body.
 12. The non-transitorycomputer-readable recording medium according to claim 10, the methodfurther comprising capturing the first sensor data using a sensor. 13.The non-transitory computer-readable recording medium according to claim10, wherein the positioning information includes either latitude orlongitude of a location corresponding to the first moving body.
 14. Thecomputer-implemented method according to claim 7, wherein thepositioning information includes either latitude or longitude of alocation corresponding to the first moving body.
 15. Thecomputer-implemented method according to claim 7, the method furthercomprising: capturing the first sensor data using a sensor.
 16. Thecomputer-implemented method according to claim 7, wherein thepositioning information includes either latitude or longitude of alocation corresponding to the first moving body.